Skip to content

Conversation

@cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Jan 22, 2026

in_tail plugins will fail to start when exceeded the number of placeholders for deleting inodes on DB.
So, this PR introduces an offloading operation which creates temp table and runs deletions with it.

Closes #11272


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
  http_server: on
  http_listen: localhost
  http_port: 2020
  health_check: on

pipeline:
  inputs:
    - name: tail
      path: /tmp/flb_manyfiles/*
      tag_regex: (/tmp/flb_manyfiles/*)(?<original_path>.*)
      tag: tail:<original_path>
      read_from_head: true
      skip_long_lines: true
      ignore_older: 1d
      db: offsets.db
      db.locking: true
      inotify_watcher: false
      mem_buf_limit: 256MB

  filters:
    - name: rewrite_tag
      match: tail:*
      rule: $target_path ^(.*)$ $1 false

  outputs:
    - name: file
      match: '*'
      path: output/path
      mkdir: true
      format: template
      template: '{log}'
  • Debug log output from testing the change

This change can detect this type of overflow and use temp database to drop the definitions beyond the number of limit of SQLite's placeholders.

[2026/01/22 15:50:28.415157022] [debug] [input:tail:tail.0] 60000 new files found on path '/tmp/flb_manyfiles/*'
[2026/01/22 15:50:28.415179297] [ warn] [input:tail:tail.0] db: large file set detected (60000 files) exceeds SQLite variable limit (32766); using temp-table cleanup for stale inode deletion
[2026/01/22 15:50:28.507476246] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@coderabbitai
Copy link

coderabbitai bot commented Jan 22, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

in_tail: Fluent Bit fails to start when monitoring a large number of files

2 participants